Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Draft] Network retriever module #3374

Merged
merged 25 commits into from
Dec 2, 2023
Merged

[Draft] Network retriever module #3374

merged 25 commits into from
Dec 2, 2023

Conversation

evan-schott
Copy link
Contributor

@evan-schott evan-schott commented Nov 3, 2023

Motivation

  • We want a richer dependency resolution system in Leo. Leo users should be able to import programs from chain/local file system as opposed to requiring .leo source files.

Components:

  1. Specify import configurations in program.json
  2. leo build leverages retriever module to pull programs from network/FS
  3. Generate leo.lock file to house the dependency configuration
  4. disassembler.rs to convert Aleo instructions into valid stub
  5. Type checking modification to maintain soundness
  6. Codegen modification to print nested imports properly

Retriever Design

Modifications to ~/.aleo file structure

  • .aleo/registry:

    • .aleo/registry/git : Where .aleo files from github.com are stored
    • .aleo/registry/testnet3: Where .aleo files from the network are stored
    • Note that new folders can be added for different networks like ./mainnet
  • Suppose building leo project dex and call leo build with dex/program.json set to:

{
  "program": "dex.aleo",
  "version": "0.0.0",
  "description": "",
  "license": "MIT",
  "dependencies": [
    {
	  "name": "credits.aleo",
	  "location": "network",
		"network": "testnet3"
    },
		{
		"name": "foo.aleo",
	  "location": "git",
		"url": "https://github.com/alice/foo.git"
		"branch": "bar"
		"rev": "3ab873bd89de"
		},
    {
	  "name": "board.aleo",
	  "location": "local",
	  "path": "/User/bob/home/work/board", 
    },
  ]
}

Specification and example flow

  1. Retrieve imports based on configurations specified in dex/program.json:
    • Must keep track of nested programs trying to build, and throw error for circular dependencies
    • Location variations:
    • "location": "network":
      • Check if credits.aleo cached already in ~/.aleo/registry/testnet3/credits.aleo
    • Retrieve from network:
      • Store copy in `~/.aleo/registry/testnet3/credits.aleo``
      • Update leo.lock
      • "location": "local": TODO
        • Build the directory specified (ex "path": "~/work/board") if no aleo program in ~/work/board/build/main.aleo
        • Other wise use ~/work/board/build/main.aleo
        • Cache in .aleo and update leo.lock
      • "location": "git": TODO
        • Example: "url": "<https://github.com/alice/foo.git>"
        • Check if board.aleo cached already as ~/.aleo/registry/git/pranav/foo/board.aleo
        • Otherwise retrieve from network and cache
        • Allow them to specify based on commit + branch
  2. Move a copy of the retrieved import(s) into dex/build/imports/
  3. Convert all imported programs to AST before start parsing main.leo

Suggested user behavior

  • Ideal: (in future)
    • User writes leo add credits.aleo kicks off network retrieval + caching process
    • Default to network for simplicity, but ability to use git or local too
  • For now:
    • User manually edits program.json
  • Its ok if user manually edits {project}/build/imports/ as that is regenerated every time build is invoked.
  • Assume that user will not modify ~/.aleo/ & leo.lock

Future features

  • Add support for local and git dependencies
  • Relax shadowing:
    • Should be able to import two functions of same name from different programs
  • Leo add {program} CLI

@evan-schott evan-schott changed the title [Draft PR] Network retriever module [Draft] Network retriever module Nov 4, 2023
@evan-schott evan-schott force-pushed the feat/stubs-disassembler branch from 087f511 to 5b0459a Compare November 4, 2023 02:14
@evan-schott evan-schott force-pushed the feat/network-retriever branch from 6982c5a to b1c9e93 Compare November 4, 2023 18:40
compiler/passes/src/type_checking/check_program.rs Outdated Show resolved Hide resolved
leo/cli/commands/build.rs Outdated Show resolved Hide resolved
@@ -42,8 +42,53 @@ impl<'a> CodeGenerator<'a> {
program_string.push('\n');
}

// Import stub programs
program_string.push_str(&input.stubs.values().map(|stub| self.visit_stub(stub)).join("\n"));
// Create directed graph representation of program dependencies.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could the Retriever module insert the stubs in the appropriate order?

errors/src/errors/cli/cli_errors.rs Outdated Show resolved Hide resolved
errors/src/errors/parser/parser_errors.rs Show resolved Hide resolved
@evan-schott evan-schott force-pushed the feat/stubs-disassembler branch from 5b0459a to c94c2c3 Compare November 30, 2023 23:49
Base automatically changed from feat/stubs-disassembler to feat/stubs December 1, 2023 17:36
@evan-schott evan-schott force-pushed the feat/network-retriever branch from 574ae0a to db20544 Compare December 2, 2023 00:21
Copy link
Collaborator

@d0cd d0cd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@d0cd d0cd merged commit 565079c into feat/stubs Dec 2, 2023
4 of 14 checks passed
@d0cd d0cd deleted the feat/network-retriever branch December 2, 2023 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants